# 4-bit quantization inference

The 4-bit AWQ quantized version of the DeepSeek-R1-0528 671B model, suitable for use on high-end GPU nodes

A 4-bit quantized version of the Gemma 3 12B model, suitable for the MLX framework and supporting efficient text generation tasks.

This is a model that uses the GPTQ method to perform 4-bit quantization on GLM-4-32B-0414, suitable for consumer-grade hardware.

Gemma 2 27B IT is a 4-bit large language model based on AutoAWQ quantization, suitable for dialogue and instruction-following tasks.

The AWQ 4-bit quantization version of QwQ-32B-Preview significantly reduces memory usage and computational requirements, making it suitable for hardware deployment with limited resources.

A medical Q&A model fine-tuned on the MedQuAD dataset based on Llama-2-7b-chat

The 4-bit quantized version of Falcon-7B-Instruct, quantized using the AutoGPTQ tool, suitable for efficient inference in resource-constrained environments.